2 UCB 1 Policy for Multi - arm Bandit Problem

To quickly recap the setup: we have 1, · · · , n arms and for each arm i, we have an associated random variable Xi bounded in [0, 1] and our goal is to minimize regret (maximize reward) where regret is defined wrt the best fixed strategy a posteriori. Every time we play arm i, the reward is picked independently and randomly with distribution Xi. We have shown in previous lectures that if we restrict ourselves in the adversarial setting, we can do at best O( √ nT log T ) so we would like to switch gears to look at stochastic inputs to minimize expected regret. The highlight of our UCB 1 policy is that we will have an almost ”to good to be true” upper bound of O(log T ). To motivate the UCB-1 policy, let’s consider the following naive strategy which will achieve O(nT 2 3 lnnT ) regret.